The present invention relates to the field of electronic systems having a video and/or audio decompression and/or compression device, and is more specifically directed to sharing a memory interface between a video and/or audio decompression and/or compression device and another device contained in the electronic system.
The size of a digital representation of uncompressed video images is dependent on the resolution and color depth of the image. A movie composed of a sequence of such images, and the audio signals that go along with them, quickly become large enough so that, uncompressed, such a movie typically cannot fit entirely onto a conventional recording medium such as a Compact Disc (CD). It is now also typically prohibitively expensive to transmit such a movie uncompressed.
It is therefore advantageous to compress video and audio sequences before they are transmitted or stored. A great deal of effort is being expended to develop systems to compress these sequences. Several coding standards currently in use are based on the discrete cosine transfer algorithm including MPEG-1, MPEG-2, H.261, and H.263. (MPEG stands for “Motion Picture Expert Group”, a committee of the International Organization for Standardization, also known as the International Standards Organization, or ISO.) The MPEG-1, MPEG-2, H.261, and H.263 standards are decompression protocols that describe how an encoded bitstream is to be decoded. The encoding can be done in any manner, as long as the resulting bitstream complies with the standard.
Video and/or audio compression devices (hereinafter “encoders”) are used to encode the video and/or audio sequence before it is transmitted or stored. The resulting bitstream is decoded by a video and/or audio decompression device (hereinafter “decoder”) before the video and/or audio sequence is displayed. However, a bitstream can only be decoded by a decoder if it complies with the standard used by the decoder. To be able to decode the bitstream on a large number of systems, it is advantageous to encode the video and/or audio sequences in compliance with a well accepted decompression standard. The MPEG standards are currently well accepted standards for one-way communication. H-261, and H.263 are currently well accepted standards for video telephony.
Once decoded, the images can be displayed on an electronic system dedicated to displaying video and audio, such as television or a Digital Video Disk (DVD) player, or on electronic systems where image display is just one feature of the system, such as a computer. A decoder needs to be added to these systems to allow them to display compressed sequences, such as received images and associated audio, or ones taken from a storage device. An encoder needs to be added to allow the system to compress video and/or audio sequences, to be transmitted or stored. Both need to be added for two-way communication such as video telephony.
A typical decoder, such as an MPEG decoder 10 shown in FIG. 1a, contains video decoding circuit 12, audio decoding circuit 14, a microcontroller 16, and a memory interface 18. The decoder can also contain other circuitry depending on the electronic system in which the decoder is designed to operate. For example, when the decoder is designed to operate in a typical television, it will also contain an on-screen display (OSD) circuit.
FIG. 1b shows a better decoder architecture, used in the STi3520 and STi3520A MPEG Audio/MPEG-2 Video Integrated Decoder manufactured by ST Microelectronics, Inc., Carrollton, Tex. The decoder has a register interface 20 instead of a microcontroller. The register interface 20 is coupled to an external microcontroller 24. The use of a register interface 20 makes it possible to tailor the decoder 10 to the specific hardware with which the decoder 10 interfaces, or to change its operation without having to replace the decoder by just reprogramming the register interface. It also allows the user to replace the microcontroller 24, to upgrade or tailor the microcontroller 24 to a specific use, by just replacing the microcontroller and reprogramming the register interface 20, without having to replace the decoder 10.
The memory interface 18 is coupled to a memory 22. A typical MPEG decoder 10 requires 16 Mbits of memory to operate in the Main Profile at Main Level mode (MP at ML). This typically means that the decoder requires a 2 Mbyte memory. Memory 22 is dedicated to the MPEG decoder 10 and increases the price of adding a decoder 10 to the electronic system. In current technology, the cost of this additional dedicated memory 22 can be a significant percentage of the cost of the decoder.
An encoder also requires a memory interface 18 and dedicated memory. Adding the encoder to an electronic system again increases the price of the system by both the price of the encoder and its dedicated memory.
FIG. 1c shows a conventional decoder inserted in a computer architecture. A conventional computer generally includes a peripheral bus 170 to connect several necessary or optional components, such as a hard disk, a screen, etc. These peripherals are connected to bus 170 via interfaces (e.g., a display adapter 120 for the screen) which are provided directly on the computer's motherboard or on removable boards.
A Central Processing Unit (CPU) 152 communicates with bus 170 through an interface circuit 146 enabling a main memory 168 of the computer to be shared between CPU 152 and peripherals of bus 170 which might require it.
The decoder 10 is connected as a master peripheral to bus 170, that is, it generates data transfers on this bus without involving CPU 152. The decoder receives coded or compressed data CD from a source peripheral 122, such as a hard disk or a compact disk read only memory (CD-ROM), and supplies decoded images to display adapter 120. Recent display adapters make it possible to directly process the “YUV” (luminance and chrominance) image data normally supplied by a decoder, while a display adapter is normally designed to process “RGB” (red, green, blue) image information supplied by CPU 152.
Display adapter 120 uses memory 12-1 for storing the image under display, which comes from the CPU 152 or from the decoder 10. A conventional decoder 10 also uses dedicated memory 22. This memory is typically divided into three image areas or buffers M1 to M3 and a buffer CDB where the compressed data are stored before they are processed. The three image buffers respectively contain an image under decoding and two previously decoded images.
FIG. 1d illustrates the use of buffers M1 to M3 in the decoding of a sequence of images T0, P1, B2, B3, P4, B5, B6, P7. I stands for a so-called “intra” image, whose compressed data directly corresponds to the image. P stands for a so-called “predicted” image, the reconstruction of which uses pixel blocks (or macroblocks) of a previously decoded image. Finally, B stands for a so-called “bidirectional” image, the reconstruction of which uses macroblocks of two previously decoded images. The intra and predicted images are likely to be used to reconstruct subsequent predicted and bidirectional images, while the bidirectional images are not used again.
Images T0 and P1 are respectively stored in buffers M1 and M2 during their decoding. The filling and the emptying of a buffer in FIG. 1d are indicated by oblique lines. The decoding of image P1 uses macroblocks of image 10. Image T0, stored in buffer M1, is displayed during the decoding of image B2, this image B2 being stored in buffer M3. The decoding of image B2 uses macroblocks of images T0 and P1. Image B2 is displayed immediately after image 10. As the locations of buffer M3 become empty, they are filled by decoded information of image B3. The decoding of image B3 also uses macroblocks of images T0 and P1. Once image B3 is decoded, it is displayed immediately, while image P4 is decoded by using macroblocks of image P1. Image P4 is written over image 10 in buffer M1 since image 10 will no longer be used to decode subsequent images. After image B3, image P1 is displayed while buffer M3 receives image B5 under decoding. The decoding of image B5 uses macroblocks of images P1 and P4. Image P1 is kept in buffer M2 until the decoding of image B6, which also uses macroblocks of images P1 and P4, and so on.
Referring again to FIG. 1c, when any component needs access to the main memory 168 either to read from or write to the main memory 168, it generates a request which is placed on the bus 170. When the request is a write, the data to be written is also placed on the bus 170. The request is processed and the data is then either written to or read from the main memory 168. When data is read from the main memory 168, the data is now placed on the bus and goes to the component that requested the read.
There are typically many components in the computer systems that may require access to the main memory 168, and they are typically all coupled to the same bus 170, or possibly to several buses if there are not enough connectors on one bus to accommodate all of the peripherals. However, the addition of each bus is very expensive. Each request is typically processed according to a priority system. The priority system is typically based on the priority given to the device and the order in which the requests are received. Typically, the priority system is set up so no device monopolizes the bus, starving all of the other devices. Good practice suggest that no device on the bus require more than approximately 50% of the bus's bandwidth.
The minimum bandwidth required for the decoder 10 can be calculated based on the characteristics and desired operation of the decoder. These characteristics include the standard with which the bitstream is encoded to comply, whether the decoder is to operate in real time, to what extent frames are dropped, and how the images are stored. Additionally, the latency of the bus that couples the decoder to the memory should be considered.
If the decoder does not operate in real time, the decoded movie would stop periodically between images until the decoder can get access to the memory to process the next image. The movie may stop and wait quite often between images.
To reduce the minimum required bandwidth and still operate in real time, the decoder 10 may need to drop frames. If the decoder 10 regularly does not decode every frame, then it may not need to stop between images. However, this produces very poor continuity in the images. This is problematic with an image encoded to the MPEG-1 or MPEG-2 standards, or any standard that uses temporal compression. In temporal (interpicture) compression, some of the images are decoded based on previous images and some based on previous and future images. Dropping an image on which the decoding of other images is based is unacceptable, and will result in many poor or even completely unrecognizable images.
The computer can also contain both a decoder and encoder to allow for video telephony, as described above. In this case, not operating in real time would mean that the length of time between the occurrence of an event such as speaking at one end of the conversation until the event is displayed at the other end of the conversation—is increased by the time both the encoder and then the decoder must wait to get access to the bus and the main memory. Not being able to operate in real time means that there would be gaps in the conversation until the equipment can catch up. This increases the time needed to have a video conference, and makes the conference uncomfortable for the participants.
One widely used solution to allow a component in a computer system to operate in real time is to give the component its own dedicated memory. Thus, as shown in FIG. 1c, the decoder 10 can be given its own dedicated memory 22, with a dedicated bus 26 to connect the decoder 10 to its memory 22. The dedicated memory 22 significantly increases the cost of adding a decoder 10 to the computer. A disadvantage of a computer equipped with a conventional decoder is that it has a non-negligible amount of memory which is unused most of the time.
Indeed, memory 22 of the decoder is only used when decoded images are being viewed on the computer screen or need to be encoded, which amounts to only a fraction of the time spent on a computer. This memory—inaccessible to the other peripherals or to the CPU—has a size of 512 Kbytes in an MPEG-1 decoder and Mbytes in an MPEG-2 decoder. Further, this memory is oversized, since it is obtained by using currently available memory components.